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ABSTRACT- 



IMMEX is a technology-based learning and assessment tool, designed to integrate 
curricular content and problem-solving skills into real world scenarios. IMMEX inherits 
much of it’s structure from case-based and problem-based learning models and provides 
teachers with quantifiable, visual feedback on student problem-solving performances so 
that interventions can be acutely informed and immediate. While most teachers utilize 
the most basic performance indicators provided by IMMEX for measuring student 
progress, richer analytic tools have shown performance nuances such as 1) significant 
correlations between problem-solving and (some) standardized test scores, 2) evidence 
that males and females approach problem-solving differently, and 3) strategic trends in 
problem-solving. The classroom integration of technology is supported by extensive 
professional development activities • including workshops where teams of teachers, 
students, and university faculty author IMMEX cases. 

www.immex.ucla.edu 
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Case-based and problem-based learning have been used as effective instructional 
strategies in professional schools for several decades (Elstein 1993; Kolodner 1993). 
These approaches have been attractive and effective not only for their inherent 
motivational factors and realistic contexts but also for the unique perspective they 
provide about student learning (Barrett and Depinet 1991). While these approaches also 
align with reform movements in math and science (See for example: Goals 2000 (1994); 
The National Commission on Excellence in Education (1983); American Association for 
the Advancement of Science [AAAS], Project 2061 (1993); National Research Council 
(1996)), the incorporation of case-based learning into the K-12 curriculum has been 
infrequent, particularly in schools with high student diversity. This is mainly due to the 
extensive and systemic curricular reorganization needed before implementation of such 
learning approaches can become widespread. The challenges in this reorganization 
process are physical (the construction of the materials), logistical (implementation and 
integration schemes), and organizational (re-aligning curricula, professional development 
and evaluation approaches); i.e. the elements in the diffusion of most innovations 
(Valente 1995; Friedman et. al.) 

Computer technologies hold promise for accelerating the adoption of alternative learning 
and assessment modes in education. Twelve years ago IMMEX (Interactive Multi- 
Media Exercises) was developed for teaching and assessing the diagnostic skills of 
medical students (Stevens 1991). IMMEX software is a set of problem-solving authoring 
and performance data collection and analysis tools, which inherits much of its theory 
from case-based (Schank, 2000) and problem-based learning. Through vigorous 
community engagement activities and extensive professional development workshops, 
these software tools have been successfully, and widely, incorporated into primary and 
secondary school classrooms (Palacio-Cayetano et. al. 1999). 

Overview of IMMEX 

Authoring teams made up of teachers, students and university faculty, use IMMEX 
software to develop snapshots of complicated and dynamic systems, in both science and 
non-science subjects, that pose situations for students to resolve by viewing and causally 
linking data. The data variables are loosely structured as menu items, or button links, and 
can be selected in any sequence. These items, and the problems posed, constitute what is 
called the “problem space” that defines the playing field for students and researchers. 

Each case opens with a prologue and students use the embedded information to frame the 
problem and to begin forming hypotheses. Then, depending on prior knowledge and 
experience, they begin to search the problem space for items that support or refute their 
current theory. To align these tasks with student abilities, these spaces, which contain 
many solution paths, have a finite structure emphasizing the development of linkages 
among problem space variables. Most cases also contain a solution to the problem. 
While Hart (1994) and others have expressed concerns regarding problems with single 
solutions, our five-years of experiences shows no evidence from student performances, 
notes associated with problem-solving or survey data, that students are viewing the 
problem-sets through the small lens of “a single right answer”. This, in part, may be due 
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to the broad perspective of a problem space afforded by IMMEX, and by the large 
number of multiple instances, or clones (sometimes as many as 60), of each case within 
the different problem-sets. 

We, and others, have also learned that for software to be meaningfully integrated and 
aligned with learning goals, strong professional development supports are a vital part of 
the educational technology equation (National Research Council 1991). Also, Johnson 
(1999) asserts that in order for technology to have a positive impact on student 
achievement, computer use must not be a separate program that students attend only one- 
half hour a week. 

To address these multiple educational dimensions, the IMMEX Project is developed 
around three broad frameworks. 

The first framework describes the structure of extensive inservice and preservice 
professional development programs focusing on the creation of case-based software by 
teams of teachers and academic professionals. The cases are relevant and often real- 
world tasks that are appropriate for, and engage the majority of students. To date, over 
400 teachers have participated in these, and other professional development activities and 
have created hundreds of cases for their classrooms. The development process is guided 
and validated by close associations with content experts and different professional 
societies (American Association of Immunologists, American Physical Society) so that 
state and national standards alignment is not only achieved, but in many cases, surpassed. 

The details of the case development process (including pedagogical issues), the definition 
of problem spaces, identification of learning goals, standards alignment, etc. are 
discussed by Palacio-Cayetano, et al. (manuscript in preparation). Evaluation studies 
have indicated that students (and teachers) perceive IMMEX as a way of fostering the 
integration of information, rather than the learning of facts. Therefore, providing 
students with core content prior to using the problem-sets, has shown to be the most 
effective classroom practice. Each case, however, is complete in that all the information 
needed to solve it is embedded within the problem space including extensive library 
resources and explanatory aids. 

The development of curricular software that encompass broad learning, assessment, and 
research goals is complex and must address the concerns of multiple audiences including 
students, parents, teachers, administrators, educators, policy makers, etc. The second 
IMMEX framework helps define classroom implementation and curricular integration 
schemes that will maximize student learning and scaffold teachers in their use of 
technology and case-based reasoning. This framework also helps refine in cost/benefit 
terms, how the integration of curriculum into technology, with the full range of supports, 
can enrich the curricular priorities for schools and districts. 

The third framework defines the role of student performance data in the discovery of 
learning paths of students in a case-based reasoning environment. Three-levels of 
reporting tools provide teachers and researchers continual performance information as 
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well as cognitive and metacognitive insights into their students thinking. The elements of 
this framework are the main focus of this paper and are summarized in Figure 1 . 



IMMEX Online Performance/Progress Reporting Level 1: 

Classroom and Individual Student 
Progress Measures 

Description: 

The cumulative class performance plots the percent of 
IMMEX problems solved against the number of problems 
performed. Teachers can follow the case solved rate over 
time. 

Individual student progress is measured by plotting the 
Performance Index (a number calculated by plotting the 
number of problems solved against the Percent solved) 
by the total number of problems solved. Peaks and valleys 
demonstrate where students are having successes and 
difficulties respectively. 

Uses: 

Immediate feedback on class and student performance. 

Links problem-solving efficiency to other metrics (AP, grades). 
Helps frame research/equity issues (gender, language, etc.). 
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(Figure 1.) 



Tracing the Development of Problem Solving Skills 

We have developed multiple measures from the rich data generated in IMMEX cases that 
enable teachers, students and educators to follow individual student and classroom 
progress at different degrees of detail. For instance, a teacher who assigns cases as out- 
of-class activities would like to know on a daily basis which students are performing the 
cases and how well they are doing (proficiency). Teachers may also want a quick check 
to compare differences in performances between classes. These proficiency measures 
constitute the Level 1 of IMMEX data reporting (Figure 1). Online, teachers can view a 
graph plotting the number of problems completed vs. the number solved and further 
scroll down to look at this ratio for individual students. They can then hyperlink to 
individual students and view individual student progress throughout the course, seeing 
what problem-sets they struggle with (drops in the curve) and those where they enjoy 
success (peaks). 

While these reporting tools are useful for teachers they can also contribute to the research 
base by providing a performance measure for correlation studies. For instance, in two AP 
chemistry classes, the correlations between problem-solving performance and prior 
STAR science tests are low, but are better for course grades and AP scores (Table 1). 
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Problem Solving Correlations 





Percent 

Solved 

(Year-long) 


Percent 


Pearson Correlation 


1.000 


Solved 


Sig. (2-tailed) 




(Year-long) 


N 


55 


STAR 


Pearson Correlation 


.323* 


Reading 


Sig. (2-tailed) 


.018 




N 


53 


STAR Math 


Pearson Correlation 


.274* 




Sig. (2-tailed) 


.047 




N 


53 


STAR 


Pearson Correlation 


.114 


Science 


Sig. (2-tailed) 


.417 




N 


53 


Mid-Term 


Pearson Correlation 


.558*’ 


Grade 


Sig. (2-tailed) 


.000 




N 


46 


Multiple 


Pearson Correlation 


.445** 


Choice 


Sig. (2-tailed) 


.001 


Exam 


N 


52 


Overall 


Pearson Correlation 


.389** 


Class 


Sig. (2-tailed) 


.004 


Grade 


N 


53 


Advanced 


Pearson Correlation 


.584** 


Placement 


Sig. (2-tailed) 


.000 


Score 


N 


53 


Percent 


Pearson Correlation 


.316* 


Solved 


Sig. (2-tailed) 


.027 


(Exam) 


N 


49 



*. Correlation is significant at the 0.05 level (2-tailed). 
**. Correlation is significant at the 0.01 level 



(Table 1. Correlation of Problem-Solving Efficiency (% of cases solved) With 
Other Performance Indicators.) 

We have also seen gender differences emerge out cumulative percentage data collected 
from a yearlong, homework-based, implementation plan. Males and females performed 
the same number of problems (Males = 36 ± 14 Females = 42 ± 11), and solved the same 
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number of problems (Males = 61 ± 8% Females = 57 ± 12%) however, the correlation 
between course grades and problem-solving proficiency was greater for girls (Males .235 
p=.221, Females .61, p=.002). In fact, the course grades and the IMMEX problem- 
solving performance account for nearly 70% of the variance in females AP performance. 
Males were more variable in their performances, with grades and problem solving only 
accounting for 40% of the AP score variance. These gender affects are explored further 
in Paek et al. (manuscript in preparation). 

Level 1 reporting features are therefore useful for providing a quick snapshot of student 
performance. Often, however, teachers wish to dig deeper into the foundations of their 
students thinking and the search path mapping features of Level 2 data reporting equip 
teachers with the tools to do so. 

While students engage in problem solving, the sequence of their actions is recorded into a 
log from which we generate visual maps of their search through each problem space. 
These maps visually display every data item selected by the student (represented by the 
color-coded rectangles arranged by content, concepts or types), the sequence in which the 
information was selected (indicated by the line traveling from the left corner to the center 
of the rectangle), and the time spent on each concept. (See Figure 2). Because these 
maps are rich in information, teachers can, and do use them in multiple ways. For 
teachers who are authors, these maps provide a validity check to ensure that the case is 
performing as intended. Also, classroom interventions are suggested by group maps that 
compare the strategies the students who solve or miss the case. 



IMMEX Online Performance/Progress Reporting Level 2: 




Search Path Mapping of Student Strategies 

Description: 

iyiEnaiwraai i la 

— Displays each test selection chosen during a case as 

a colored rectangle and connects the sequence with 
a line. A literal map of a student's thinking. 

Uses: 

Visualizes student problem-solving approaches. 
Repeat performances document improvement. 
Serves as an artifact for student reflection. 

Provides a structure for scoring rubrics. 




( Figure 2.) 

By comparing earlier maps to later ones, a teacher can judge refinements of problem 
solving approaches over time as shown in Figure 2. Other teachers provide these maps 
to students to encourage reflection and have them write essays analyzing their own 
progress (Lawton 1998). Search path maps are particularly important for examining the 
more metacognitive aspects of problem-solving such as persistence, elimination of 
alternative hypotheses, efficiency, confidence and certainty, and related professional 
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development activities focus on this type of educational intervention. Lastly, these Level 
2 maps provide the artifacts for developing problem-solving scoring rubrics and for 
revealing the nature of the clustering of performances. 

Level 3 of IMMEX data reporting seeks to link search path maps with classroom 
interventions to improve learning. Examination of thousands of student performances in 
many domains and across multiple grade-levels of education (elementary through 
medical school) has indicated that many search strategies/approaches can be applied 
while solving IMMEX problems. Students differentially use these strategies, often 
unconsciously (Reder & Schnn 1996), and through this use, they can develop higher 
order strategies. 




Strategy Transition Diagrams 




(Figure 3.) 



Description: 

The "Limited" strategy group (left) is represented as the pink 
rectangles, "Prolific" (right) by the yellow, and "Efficient" (top) 
by the green. The groups are defined by scoring rubrics of 
neural network clustering and the thickness of the lines 
proportionaly indicates transitions among strategy groups 
upon repeat testing. 

Uses: 

Provides probabilistic inferences about student progress. 
Suggests points of intervention. 

Documents effects of interventions. 

Allows classroom progress comparisions 



In following these strategies over time, it has become apparent that, for each problem-set, 
there are transitional states that students may (or may not) pass through as they develop 
expertise. (Figure 3.) Level 3 data analysis demonstrates how students arrive at, and 
depart from these states enabling researchers to use this information in a predictive 
fashion. The key component of this research is the development of valid and rapid 
procedures for classifying the strategies revealed by the search path maps. Categories of 
problem-solving states are being determined through both rubrics (analysis of student 
search path maps) as well as natural clustering of key components of the problem-solving 
process by artificial neural networks. Each provides a different level of contextual 
information and granularity and can often be productively combined. 

Neural network clustering is an automated technique where the sequence of student 
actions during problem solving is used as input to unsupervised artificial neural networks 
that perform classifications based on these sequences. We have shown this approach 
applicable to both simple and complex problem-solving situations (Stevens and Najafi 
1993; Casillas et. al. 2000). 
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The second, more traditional approach involves the development of scoring rubrics by 
teams of teachers, educators, content experts and case authors. In this procedure, 
performance guidelines are first established and agreed upon based on the case and 
representative student work. Next, scoring teams use these rubrics to rate large numbers 
of search path maps and the rubrics are refined for performances where the inter-rater 
reliability is low. This approach can be guided by neural network groupings as a starting 
point for in depth performance analysis, observing problem-solving trends, and the 
transitioning between states over time. Thus rubrics incorporate “person-specific” key 
elements, while neural network provides strategic clustering without the associated 
interpretative information. 

An example by Vendlinski (2000) describes a combination of these approaches. He used 
artificial neural networks to identify common strategies in over 800 student performances 
by using a technique similar to ones described in Klahr and MacWhinney (1997). Based 
on the resulting strategies, the amount of information a student reviewed and the 
likelihood that the strategy was successful, he grouped the strategies into three general 
categories. Category 1 strategies were termed “limited” and contained performances 
where students had only gathered enough information to do little more than guess at an 
answer. For the most part, these students were unlikely to solve the problem. Category 2 
strategies were termed “prolific” and consisted of performances where students had 
accumulated more than enough information to solve the case, but were unable to convert 
the information they had into a successful solution to the problem. Category 3 strategies 
were labeled “efficient” and most often resulted in a successfully solved problem. This is 
similar to a finding by Maker (1994) that suggests, “high” competence is the ability to 
solve the most complex (or the most simple) problems in the most efficient, effective or 
economical ways.” The student performances were then ordered and plotted so 
transitions between categories could be seen. As shown in Figure 3, a similar number of 
students began with limited or with prolific strategies, while fewer students began with 
efficient strategies. Most students who begin with a limited strategy, remain there 
(despite little success at solving problems). These students appear unlikely to improve 
their problem-solving abilities without intervention. While a number of students who 
begin with prolific strategies do transit to a more effective strategy, most seem to fall 
back into using more profuse strategies in subsequent attempts to solve IMMEX cases. 
These students might also profit from problem-solving interventions. Lastly, most 
students who begin solving problems efficiently continue to use similar strategies in 
subsequent cases. These trends are even more apparent when the first strategy type 
chosen by a student is juxtaposed against the most frequent strategy type the student used 
to solve problems. As shown in Table 2, the first strategy type is significantly correlated 
with the most frequent strategy type chosen by the student. 
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Categpry 1 
strategies on most 
problems 


Category 2 
strategies on most 
problems 


Categpry 3 
strategies on most 
problems 


Category 1 
strategy on first 
problem 


30 


8 


6 


Category 2 
strategy on first 
problem 


5 


35 


8 


Category 3 
strategy on first 
problem 


5 


9 


23 



(Table 2. This table compares the category of the strategy a student used to solve his or her first qualitative 
Chemistry IMMEX problem to the category of the strategy a student used most often to solve different 
versions of the same problem. There is significant statistical evidence (% 2 = 70.5; d.f. = 4; pc.OOO) to 
conclude that this effect is not random.) 

Using this approach we can not only identify students who successfully make the 
transition from less successful to more successful strategies (and perhaps why), but also 
the probabilities that students will make such transitions without teacher intervention. 
When interventions are attempted, we can use this information to test the effectiveness of 
those interventions and individual student improvement. Such a tool also provides 
important feedback to educators and allows them to document the overall effects of their 
teaching approaches and methods. 

The power of this approach is that probabilities can now be derived for students at each 
strategy level and that interventions can be tested for their effect on changing these 
probabilities where appropriate. These diagrams also provide a means for documenting 
effects of overall classroom teaching approaches and methods. 

Conclusions: 

At a recent meeting Dr. Linda Roberts of the Department of Education remarked that 
“There is no longer an interest in focusing on “technology for technology sake”, it is not 
about the computers or the software, it is about how they are utilized for learning” (L. 
Roberts, 2000). 

The IMMEX Project, by virtue of its close teacher partnerships and comprehensive 
frameworks for developing and implementing software, is beginning to construct a model 
for the successful local production and utilization of technology-based learning systems. 

Much of the impetus for this development, however, resides in the shared understanding 
that the development of rich models of student performance and progress in different 
disciplines and at different levels of education, represents the next stage in the evolution 
of learning technologies. To learn more about The IMMEX Project implementation and 



research methodologies, to view the problem-sets available, or for contact information, 
visit www.immex.ucla.edu. 
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